New global optimization algorithms for model-based clustering

The Expectation-Maximization (EM) algorithm is a very popular optimization tool in model-based clustering problems. However, while the algorithm is convenient to implement and numerically very stable, it only produces solutions that are locally optimal. Thus, EM may not achieve the globally optimal solution to clustering problems, which can have a large number of local optima. This paper introduces several new algorithms designed to produce global solutions in model-based clustering problems. The building blocks for these algorithms are methods from the operations research literature, namely the Cross-Entropy (CE) method and Model Reference Adaptive Search (MRAS). One problem with applying these two approaches directly is the efficient simulation of positive definite covariance matrices. We propose several new solutions to this problem. One solution is to apply the principles of ExpectationMaximization updating, which leads to two new algorithms, CE-EM and MRAS-EM. We also propose two additional algorithms, CE-CD and MRAS-CD, which rely on the Cholesky decomposition. We conduct numerical experiments to evaluate the effectiveness of the proposed algorithms in comparison to classical EM. We find that although a single run of the new algorithms is slower than EM, they have the potential of producing significantly better global solutions to the model-based clustering problem. We also show that the global optimum “matters” in the sense that it significantly improves the clustering task.